Update 1: Threshold Pre-Download Functionality

Previously, the CatFIM had to hit the WRDS API for each run to download the thresholds. Now, CatFIM will automatically download and save the thresholds into a .pkl file each run.

This is similar to how the metadata.pkl file is made, but only the thresholds for the selected HUCs will be saved (unlike metadata, which downloads for all sites).

If you want to download all sites, you can use the download_all_thresholds() function to predownload all thresholds.

Update 2: New stage-based CatFIM Region - Guam

Our new ability to run CatFIM from fully downloaded data allows us to run stage-based CatFIM for Guam, which doesn't have stage data available on WRDS.

This update allows stage-based CatFIM to run with manual input data.

Note: For manual input data, stage-based CatFIM currently skips the vertical datum and elevation adjustment processes. This is valid given the assumption that manually input data will be subjected to a higher level of quality control and will already pass the USGS gage acceptance criteria. It also allows us to avoid having to recreate the USGS gages data for each new manual site.

Update 3: New geospatial data requirements

NHD streamline data for Guam and American Samoa was copied down from EC2 into OWPAL. It is the same file structure from the EC2. The nhdplus folder and everything inside of it is new to OWPAL.

/data/inputs/nhdplus/Guam_6637/NHDFlowline_Guam_6637.gpkg
/data/inputs/nhdplus/AmericanSamoa_32702/NHDFlowline_AmericanSamoa_32702.gpkg

Update 4: Code for pre-processing Guam input data

Thresholds data for Guam # TODO: Fill in
Metadata for Guam # TODO: Fill in

Changes

• tools/catfim/generate_categorical_fim.py: Added the option to provide an input thresholds file (rather than hitting the WRDS API). Added functionality to skip elevation adjustment for manual inputs. Added functionality to process two additional regional input files. Added docstrings for all functions.
• tools/catfim/generate_categorical_fim_flows.py: Added functionality to process two additional regional input files. Created the __load_thresholds() function to manage getting the thresholds from the WRDS API or the input thresholds file. Added docstrings for all functions.
• tools/catfim/generate_categorical_fim_mapping.py: Added docstrings for all functions.
• tools/tools_shared_functions.py: Updated the get_thresholds() function to produce a status message (and took out the threshold_count output). Created the download_all_thresholds() function that predownloads all thresholds for a list of AHPS LIDs.
• data/nws/preprocess_ahps_nws.py: Changed output to get_thresholds() function.
• data/usgs/preprocess_ahps_usgs.py: Changed output to get_thresholds() function.

Testing

Generally, you do not copy this part into the ChangeLog. These are some quick notes on what you did test and/or notes for the reviewer to help with their review testing.

Deployment Plan (For FIM developers use)

• Does the change impact inputs, docker or python packages?

  • Yes
  • No (f no.. skip the rest of the Deployment Plan section)

• If you are not a FIM dev team member: Please let us know what you need and we can help with it.

• If you are a FIM Dev team member:

  • Please work with the DevOps team and do not just go ahead and do it without some co-ordination.

  • Copy where you can, assign where you can not, and it is your responsibility to ensure it is done. Please ensure it is completed before the PR is merged.

  • Has new or updated python packages, PipFile, Pipefile.lock or Dockerfile changes? DevOps can help or take care of it if you want. Just need to know if it is required.

    • Yes
    • No

  • Require new or adjusted data inputs? Does it have a way to version (folder or file dates)?

    • No
    • Yes
      • Require new pre-clip set or any other data reloads, such as DEMS, osm, etc. ie.. pre-requisite re-data upstream of your input changes.
        • Yes
        • No
      • Has the inputs been copied/exist in all five enviros:
        • FIM EFS
        • FIM S3
        • ESIP
        • Dev1
        • UCS2

• Please use caution in removing older version unless it is at least two versions ago. Confirm with DevOps if cleanup might be involved.

• If new or updated data sets, has the FIM code, including running fim_pipeline.sh, been updated and tested with the new/adjusted data? You can dev test against subsets if you like.

  • Yes

Notes to DevOps Team or others:

Please add any notes that are helpful for us to make sure it is all done correctly. Do not put actual server names or full true paths, just shortcut paths like 'efs..../inputs/, or 'dev1....inputs', etc.

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations